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Abstract. We analyze the storage eapacity of the Hopfield models on classes 
of random graphs. While such a setup has been analyzed for the case that the 
underlying random graph model is an Erdos-Renyi-graph, other architectures, 
including those investigated in the recent neuroscience literature, have not 
been studied yet. We develop a notion of storage capacity that highlights the 
influence of the graph topology and give results on the storage capacity for not 
too irregular random graph models. The class of models investigated includes 
the popular power law graphs for some parameter values. 



1. Introduction 

Thirty years ago, in 1982, Hopfield introduced a toy model for a brain that 
renewed the interest in neural networks and has nowadays become popular under 
the name Hopfield model [H]. This model in its easiest version assumes that 
the neurons are fully connected and have Ising-type activities, i.e. they take the 
values +1, if a neuron is firing and —1, if it is not, and is based on the principles 
of statistical mechanics. Since Hopfield's ground-breaking work it has stimulated 
a large number of researchers from the areas of computer science, theoretical 
physics and mathematics. 

In the latter field the Hopfield model is particularly challenging, since it also 
can be considered as a spin glass model and spin glasses are notoriously difficult 
to study. A survey over the mathematical results in this area can be found in 
either |1] or [20] . It is worth mentioning, that even in the parameter region where 
no spin glass phase is expected the Hopfield model still has to offer surprising 
phenomena such as in [T3] . 

When being considered as a neural network one of the aspects that have been 
discussed most intensively is its so-called storage capacity. Here, one tries to 
store information, so-called patterns in the model, and the question is, how many 
patterns can be successfully retrieved by the network dynamics, i.e. how much 
information can be stored in a model of neurons. One of the early mathematical 
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results states, that if the patterns are independent and identically distributed 
(i.i.d. for short) and consist of i.i.d spins and if their number M is bounded by 



true, if one starts with a corrupted input - if more than fifty percent of the input 
spins are correct, one still is able to restore the originally "learned" patterns. 
However, if one also allows for small errors in the retrieval of the patterns one 
obtains a storage capacity of M = aN for some value of a smaller than 0.14 
(see [25], [18], [22] )• This latter result is in agreement with both, computer 
simulations as well as the predictions of the non-rigorous replica method from 
statistical physics (see [T]). 

The setup of the Hopfield model has been generalized in various aspects, e.g. 
the condition of the independence has been relaxed (see [IH], [22]), patterns with 
more than two spins values have been considered (see [12] , [20] , [21] ) , and Hopfield 
models on Erdos-Renyi-graphs were studied ([5], [6], [22], [23]). The present paper 
starts with the observation that, even though being more general than the com- 
plete graph, also Erdos-Renyi-graphs do not seem to be the favorite architectures 
for a brain for scientists working in neurobiology. There, the standard paradigm 
currently is rather to model the brain as a small world graph (see [2], [2H])- We 
will focus on the question, how many patterns can be stored in a Hopfield model 
on a random graph, if this graph is no longer necessarily an Erdos-Renyi-graph. 
There is a major influence of the graph structure on the model's capability to re- 
trieve corrupted information. The relationship between network connectivity and 
the performance of associative memory models has already been investigated in 
computer simulations (see for instance [7]). Therefore the goal of the present note 
is to establish rigorous bounds on the storage capacity of the Hopfield model on 
a wide class of random graph models, where we interpret " storage" as the ability 
to retrieve corrupted information. 

To this end we organize the paper in the following way: Section 2 introduces 
the basic model we will be working with in the present paper. It also addresses 
the question, what exactly we mean when talking about the storage of patterns. 
Section 3 contains the main result of this paper. A main ingredient of the proof 
is to analyze the spectrum of the adjacency matrix of the graph that serves 
as a model of the network architecture. This analysis is provided in Section 
4. Eventually, Section 5 contains the proof of the main result. An appendix 
will contain estimates on the minimum and maximum degree of an Erdos-Renyi 
graph. These are needed to apply our main result to the setting of such random 
graphs and may also be of independent interest. 
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2. The model 

The Hopfield model is a spin model on G N spins, a G Sjy := {— 1,+1}^ 
describes the neural activities of neurons. The information to be stored in the 
model are patterns S}, ■ ■ ■ , G { — 1, +1}^. As usual, we will assume that these 
patterns are i.i.d. and consist of i.i.d. spins (,^f ) with 

m = ±1) = \- 

Note that M may and in the interesting cases will be a function of A^. The 
architecture of the Hopfield model is an undirected graph G = {V,E), where 
V = 1, . . . ,N. With the help of the patterns and the graph one defines the 
Hamiltonian (or energy function) of the model by 

N M 



Hn{(t) = -Const. (A^) 



j 



Here 1 if the edge between i and j is in E and = otherwise. 

The constant is chosen in such a way that the mean free energy of the model is 
finite and not constantly equal to zero. The idea of this setup is that the patterns 
(as well as their negatives — {C,'^),fi = 1, . . . , M) are hopefully local minima of Hj^. 
This is easily checked, if M = 1 and G is the complete graph, since then 

TV 
i=l 

and hoped to be inherited by the more general model, as long as M is small 
enough. If this is indeed the case, then not only the patterns are points in Sat 
that have a large weight under the (random) Gibbs measure 

exp(-/3i7jv(a)) 
fiN[o-) := , p > 

(where Zn is a normalizing constant), but also they are fixed points of the parallel 
gradient descent dynamics T = (Tj) on Sat. By definition Tj only changes the 
i'th coordinate of a configuration a and 

TV M 

T{a) = {Ti{a), . . .,Tn{(t)), with Ti{a) = sgnC^ajaij^^^^^). 

j=i fi=i 

Hence Tj flips the ai if and only if this lowers the energy. The dynamics T can 
be thought of as governing the evolution of the system from an input towards 
the nearest learned pattern. being fixed point of T can thus be interpreted 
as recognizing a learned pattern. However, this is not really what one would call 
an associative memory. An important feature of the standard Hopfield model 
(the one where G = K]\f, the complete graph on A^ vertices) is thus also that 
under certain restrictions on M (and the number of corrupted neurons), with 
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high probabihty, a corrupted version of say ^'^ converges to when being 
evolved under T. This observation is also crucial for the present paper. 
When considering the stability of a random pattern C,^ under T in the above 
setting, we need to check whether Tj(^'^) = holds for any i. Now 

N M N N 

j=l u=l j=l j=l Vt^ii 

I.e. we have a signal term of strength d{i) (the degree of vertex i) and a random 
noise term. The first observation is that the network topology enters via the 
degrees of the nodes. Indeed in such a simple setup - the stability of stored 
information - the minimum degree of the vertices is clearly decisive to compute 
the models's storage capacity : in the case where a vertex i has a small degree, 
the noise term will exceed the signal term, except for a very small number of 
stored patterns. However, if seems to be obvious that also global aspects, e.g. 
whether or not the graph is connected, must play a role. This is confirmed, if we 
are stetting up a Hopfield model on graph G consisting of a complete graph Km 
(on the vertices 1, . . . , m) and the graph K^-m on the vertices m + 1, . . . , N with 
log N m N and if we assume that these two subgraphs are disconnected or 
just connected by one arc. Each of the vertices thus has at least degree m and it 
can be computed along the hues of [21] or [26] that at least 2i^n P^-tterns can be 
stored as fixed points of the dynamics T. However, if we try to store one pattern, 
e.g. with = 1 for alH = 1, . . . , A^, and start with a corrupted input with 

~i / ~1 i < m 

m + l<i< N 

we see that 

m') = e7- 

Hence is a fixed point of T implying that the retrieval dynamics is not able to 
correct m N errors, even if we just want to store one pattern. So, if we insist 
that a neural network should also exhibit some associative abilities (and this has 
always been a central argument for the use of neural networks), we have to take 
the graph topology into account. 

This topology is encoded in the so called adjacency matrix A of G. Here A = (aij) 
and ttij = 1, if Cjj G E and a^j = otherwise. If G is sufficiently regular, the 
connectivity of G (which played an important role in the above counterexample) 
can be characterized in terms of the spectral gap. To define it, let Ai > A2 > 
. . . > Atv be the (necessarily real) eigenvalues of A in decreasing order. Define k 
to be the second largest modulus of the eigenvalues, i.e. 

K := max |Ai| = max{A2, |A7v|}. 

Then the spectral gap is the difference between the largest eigenvalue and k, 
i.e. Ai — K,. However, also the degrees of the vertices are important. Hence let 
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di = Qij be the degree of vertex i. We will denote by 

6 := min di and m := maxdi 

i i 

the minimum and maximum degree of G, respectively. 

3. Results 

We will now state the main result of the present paper. 

In order to formulate it let us define the usual Hamming distance on the space 
of configurations E^v, 

dHia,a') = ^[N-ia,a')]. 

In other words, dn counts of indices where a and a' disagree. For any a E T^n 
and g G [0, 1], let S{a, g) the sphere of radius gN centered at a, i.e. 

S{a, g) = {a' ■.dH{a,a') = [gN]}, 

where [gN] denotes the integer part of gN. 



For the rest of the paper we will suppose that the following hypothesis is true : 



{Hi) There exists ci g]0, 1[, such that 6 > CiXi (recall that 6 is the minimum 
degree of the graph G, and Ai is the largest eigenvalue of its adjacency matrix). 



To understand condition (HI) recall that for a regular graph with degree d the 
largest eigenvalue of A equals d and so does its minimum degree 6. Condition 
(HI) can thus be interpreted as the requirement that G is sufficiently regular. 
Under this condition we will prove, that we can store a number M of patterns 
depending on Ai and the spectral gap of A - even in the sense that the dynamics 
T repairs a corrupted input. Mathematically speaking we show 

Theorem 3.1. With the notation introduced in Section 2, if (HI) is satisfied 
and Ai > clog(A^)K for some c > large enough, then there exists Oc > and 
Po e]0, l/2[ such that if 

M = a—\- 

m log i\ m 

for some a < ac, then that for all p G]0,po] '"'e obtain 

P[V/i = 1, . . . , M, Vx G p) : T'=(x) = ^1, as N ^oo, 

for k = (9(max{loglog A^, — ^^^17— :})■ Here is defined as the k'th iterate of 

^°S(Klog(JV) ) 

the map T. 
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In other words, Theorem 13.11 states that we are able to store the given number 
of patterns in such a way, that a number of errors that is proportional to can 
be repaired by a modest (at most 0{logN)) number of iterations of the retrieval 
dynamics. The number of patterns depends on the largest eigenvalue and the 
spectral gap of the adjacency matrix and is larger for large spectral gaps. 
Before advancing to the proof we will apply this result to some classical models 
of random and non-random graphs. 

Corollary 3.2. If G = Kj^, i.e. in the case of the classical Hopfield model, the 
storage capacity in the sense of Theorem \3.1\ is M = aj^;^ for some constant a. 
The number of steps needed to repair a corrupted input is of order 0{\og\og N) . 

Proof. The complete graph is regular, hence condition (HI) is satisfied. From 
Theorem 13.11 we obtain the numerical values for M and the number of steps by 
observing that in the case of the complete graph the eigenvalues of A are TV — 1 
and —1 (the latter being an — 1-fold eigenvalue). □ 

It should be remarked that similar results were obtained by Komlos and Paturi 

m- 

We mainly want to apply our results to some random architectures, i.e. G will 
be the realization of some random graph. The most popular model of a random 
graph is the Erdos-Renyi-graph G{N,p). Here all the possible (^) edges occur 
with equal probability p = p{N) independently of each other. Hopfield models 
on G{N,p) have already been discussed in [5], [29], or [23] . 
Here we obtain 

Corollary 3.3. If G is chosen randomly according to the model G{N,p), then, if 
p ^ for a set of realizations of G the probability of which converges to one as 
N — )■ oo, the capacity (in the above sense) of the Hopfield model is c pN/ log(A^) 
for some constant c > 0. 

Proof. For the eigenvalues of an Erdos-Renyi-graph it is well known that with 
probability converging to 1, as — )■ oo (such a statement in random graph theory 
is said to hold asymptotically almost surely), Ai = (1 + o{l))Np and k, < cy/Np 
(see for instance [22]) • Moreover, we can control the minimum and maximum 
degree in G{N,p). Indeed, for our values of p we have m = [1 + o{l))Np and 
5 = [1 + o{l))Np asymptotically almost surely. Surprisingly, we could not find 
this result in the literature and thus proved it in Appendix A. 
Hence [Hi) is satisfied. □ 

For the next example let us give a general construction of random graph models. 
To this end let io and A^ positive integers and L = {iQ,io + l,io + N — 1}. For 
a sequence w = {wi)i^L, we consider random graphs G{w) in which edges are 
assigned independently to each pair of vertices {i,j) with probability 



Pij = pWiWj, 
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where p = ^/ Ylk^h'^k- We assume that 

i ^ 

keL 

SO that pij < 1 for all i and j. It is easy to see that the expected degree of i is Wi. 
For notational convenience let d = 'Yliieb'^il^ expected average degree, 

m the expected maximum degree and 

be the so called second order average degree of the graph G{w). 
We now turn to a subclass of random graphs that have recently become very 
popular, power law graphs [11]. Power law random graphs are random graphs in 
which the number of vertices of degree k is proportional to 1/A;^ for some fixed 
exponent /3. It has been realized that this "power law" -behavior is prevalent in 
realistic graphs arising in various areas. Graphs with power law degree distribu- 
tion are ubiquitously encountered, e.g. in the Internet, the telecommunications 
graphs, the neural networks and many biological applications ([15]; [2Z],[SI])- Us- 
ing the G{w) model, we can build random power law graphs in the following way. 
Given a power law exponent /3, a maximum expected degree m, and an average 
degree d, we take Wi = ci~~ for each i G {io, . . . ,io + l,io + N — 1}, with 

/3-2 



-dN^^ 



and 



m{i3-i) 

For such power law graphs we obtain: 

Corollary 3.4. // G is chosen randomly according to a power law graph with 
/3 > 3 then, if m ^ d > c\/^{log{N)Y^'^ for some constant c > for a set 
of realizations of G the probability of which converges to one as N ^ oo, the 
capacity (in the above sense) of the Hopfield model is C'(/^)^i~(]v) ■^^'^ ^ constant 
G that only depends on p. 

Proof. By definition, if d <C m, then the minimum expected degree Wmin = 

c{io + N - 1)" A satisfies Wmin = jE^d{l + o(l)). 

From [8], we learn about the second order average degree that 

if /3 > 3. 

On the other hand, Chung and Radcliffe prove in [10] the following : if the 
maximum expected degree m satisfies m > | log(\/2iV), then with probability at 
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least 1 — we have 

Ai(GH) = (1 + o(l))J and k{G{w)) < \l mi\og{V2N) . 

From this, in particular, we can deduce that hypothesis {Hi) is satisfied. 
To apply Theorem 13.11 we need to compare m to the maximum degree m of a 
graph G, chosen randomly according to a power law graph with /3 > 3. As proven 
in [9], we have the following estimate, using Chernoff inequalities : for all c > 0, 
there exist two constants cq, ci > such that 

P[3i G L : — Wi\ > cWi] < ^^exp(— CqWi) < exp(— CiC? + log A^), 
since Wmin = 0{d). 

Under our assumption that d ^ log A^, we deduce from this estimate that m = 
C m(l + o(l)), for some C > 0, and we finally obtain that the capacity of 
the Hopfield model on power law graphs (for a sequence of sets of graphs with 
probability converging to one) is at least 

A? kXi d'^ 
const— —i— = C{(3): 



m\og{N) m m\og{N)' 

' log(Af)" 



if, /3 > 3, and k < C2|-^nr7Y for some C2 > small enough. This is true in particular. 



if 

Wlog(V2A^) < C3 ^ 



log(A^) 

for some C3 small enough, i.e. 

□ 

4. Technical preparations on random graphs 

We first present the results we will use in the proof of our theorem. Let G be a 
simple graph with A^ vertices and / edges. Recall that for such a graph 



Ai > ... > A 



N 



are the (real) eigenvalues of its adjacency matrix and k, = max{A2, |AAr|}. 
We begin with an estimate of the moment generating function of a sum of i.i.d. 
random variables, related to G. We assign i.i.d. random variables Xi to the ver- 
tices of G, taking values ±1 with equal probability. Let us define the "quadratic 
form" over G 

S = XiXj. 

The following theorem due to Komlos and Paturi gives an upper bound on the 
moment generating function of S, which appears naturally when we use an ex- 
ponential Markov inequality for an upper bound. 



ASSOCIATIVE MEMORY MODEL ON RANDOM GRAPHS 



9 



Theorem 4.1 (J. Komlos, R. Paturi [17]). The moment generating function of 
S can be bounded as 

E[e-''^]<Ele-]<eMj^^^). 

forO<t < 1/Ai. 



Corollary 4.2. For any y > 0, we have 



P[S >y]< exp( 



2{l + Xiy)' 



Proof. Apply the exponential Markov inequality together with Theorem 14.11 to 
see that ^ 

P[S >y]< e-'yE[e''] < exp{-ty + ^^^^y^), 

for < t < l/Ai- The desired estimate is obtained by the choice of t = iqif^ 
which is smaller than 1/Ai . □ 

As we will apply this result for subgraphs in the proof of our main result, we need 
also an estimate of the largest eigenvalue \i{H) of particular subgraphs H of G. 
To this end we will quote another result by Komlos and Paturi. 

Lemma 4.3 (J. Komlos, R. Paturi [I?]). Let G be a simple graph with N vertices. 
If I and J are two subsets of the vertex set of G with \I\ = pN and \ J\ = p'N, 
where p, p' G (0, 1). Then the number of directed edges e(J; /) going from J to I 
is at most 

e{J,I) < [pp'XiiG) + y^K]N. 
Moreover the largest eigenvalue (of the adjacency matrix) of the graph H deter- 
mined by the undirected edges from I to J is bounded as 

Xi{H) < 2[v^Ai(G) + (1 - v^)k(G)]. 
5. Proof of the main result 

We are now ready to begin with the Proof of Theorem I3.1[ We first present an 
important lemma that determines the behavior of the system for one step of the 
synchronous dynamics, more precisely it controls, how many errors are corrected 
by one step of the dynamics. 

Lemma 5.1. Let ^ 

If M < cAi for some constant a > 0, there exists po G (0, |) and constants 
Ci, C2 > 0, such that for all p G [pm, Po] we have: 

P[\fp G {1, . . . ,M}, Vx G Sie,p) : dH{T{x),e) < /(p) N] > 1 - e^, 
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where 

K K Mk 1 

f{p) =max{cip {—f,ciph{p),ci—h{p),cip (— log(-))^/^ p„} < p, 
Ai Ai (Ai)^ p 

ej\f > 0, En as N +oo and 

Hp) = -piogp - (1 - p) log(l - p) 
is the entropy function. 



Proof. We will prove that 

M 

J2 P[^x e p) : dH{T{x),^n > f{p)N] < Bn (5.1) 

To simplify notations, we can assume that the fundamental memory in question 
is Let X G {—1, 1}^ such that dniC^ ,x) = pN and I be the set of coordinates 
in which x and differ. Let T{x) be the vector resulting after one step of the 
parallel dynamics, and J be the set of coordinates in which T{x) and differ. 
Now define the weight matrix W as W = (wij) and 

M 

Then for all j E J, we have ^j{Wx)j < 0, which implies ^]{Wx)j < 0. 
For later use set 

N 

jeJ k=i 

and 

M 

(1=1 jeJ 

Observe that if p < 1/2, then x tends to be closer to than to any other pattern, 
and S^{J,I) will be the dominating term in S{J,I). We will first give a lower 
bound for S^{J,I). We can rewrite S^{J,I) as 

N 

jej fc=i jej 

where again we use the notation e{J,I) and e{j,I), to denote the number edges 
going from the set J to the set J, or, respectively, from the vertex j to the set /. 
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Under the assumption of hypothesis (Hi) and with the help of Lemma 14. 3^ we 
have for all / and J, 

S'UI) >ciAi|J|-2(|/||J|^+ V^iTMl'^) 
= Ai|J|(ci-2p-2./4f), 

V p Ai 

where p' = jj-. If we assume that p' > C2P (t— )^ for some C2 > large enough, 
and p < po for some po G (0, 1/2) small enough, we get 

S\j,I)>CiXi\J\, (5.2) 

for some constant Ci G (0, 1). 
For /i > 2, we compute 



u 11 

'A 

(j,k)(iE(j,i) {j,k)eEiJ,i) 

{j,k)(^E[jy) {j,k)€E{J,I) 

where = Q^i, for alH = 1, . . . , and /i = 1, . . . , M. To apply the results for 



the moment generating function of quadratic forms introduced in Theorem 14.11 
and Corollary 14.21 , we need to rewrite these sums over ordered pairs of vertices 
as sums over unordered pairs. We have 

E{J, V) = E{J, J) + E{J, J) = 2E{J, J} + E{J, J] = E{J, V} + E{J, J}, 

where for K,L C V E{K, L) is the edges set of the directed graph between the 
sets K and L induced by our original graph. Likewise E{K^ L} denotes the 
corresponding set of undirected edges. In the same way we obtain: 

E( J, /) = E( J n 7, J n /) + E{J r]i,Jr]i) + e{j, / n J) 
= E{J n /, J n /} + 2E{J n /, J n /} + e{j, inJ} 
= E{j,i} + E{Jni,Jni} 

Eventually, 

E{J, V) - 2E{J, I) = E{J, V} - 2E{J, 1} + E{J, J} - 2E{J nl^Jf] I}. 
We want to prove that 

MP[3/, |/| = pN, 3 J, \J\ = p'N, S{J, /) < 0] — > 0, 

as iV — )■ +00. 
To this end set 



(j,k)€E{J,V} {j,k)eE{J,I} 



Sl^iJ) = E "K' and54^(J,/)= 



A- 

(j,k)eE{j,j} {j,k)eE{Jni,Jni} 
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Then 

MM MM 

sij, I) = s\j, + s^{j) - 2 Yl s^iJ^ ^) + E (^) - E ^4 n- 

11=2 fj,=2 fi=2 /i=2 

Let 7i, 72, 73, 74 > 0, such that 71 + 272 + 73 + 74 = 1- 

We will consider the four sums separately. First, using f l5.2p . we have 

M 

P[3I, \I\ = pN, 3 J, I J| = p'iV, S'liJ) < -liS\J, I)] 

/i=2 

M 

J:\J\=p'N M=2 

Given the vector C,^ = {Q)i=i,...,N, the random variables (^if )f=L''.'jv^ are condi- 
tionally independent and uniformly distributed on { — As the estimates 
we will get for the conditional probabilities and the moment generating function 
will not depend on the choice of they will be true also for the unconditional 
probabilities. 

Given the vector the random variables S'^{J), fi = 2, . . . , M, are independent. 
Similar to the estimate of Corollary 14. 2[ we obtain 

F[f;sf(j) < -7.c,A,|j|i < exp (J-n^h^ 

M=2 V ^-^ + 7iCiAi|J| 

where Aj = Xi{E{J, V}) is the largest eigenvalue of the graph determined by the 
undirected edges in E{J, V}. Using Lemma we have 

Aj < 2[v^Ai + (1 - v^)k], 

and moreover e{ J, V} < e(J, V) < m\J\ is trivially true. We deduce that 



P±Sf(J) < -„C.A,|,/|| < exp (-2^-^'^ 



Mm 



There are (|^|) ways to choose the set J, and by Stiling's formula ( J|) < exp{h{p')N), 
where h{x) = — xlogx— (1— x) log(l— x) is the entropy function introduced above. 
Using h{p') < — 2p'log(p'), we obtain the condition: 

7iCi 1 



4 2^7 + 2^ + :^^ 

which is true if 

7iC: 

as well as 

7 

8 2k 



+ log(p') >0, 



i-l= + log(p') > 0, (5.3) 
2VP^ 

^ ^ + log(p')>0. (5.4) 

71C1A1 
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Now, there exists a po ^ (0,0.1), such that the first condition fl5.3l) is true if 
p' < Pq. The second condition (15 ■4p is true if 



Ai 



p >exp(-c- ^ ^^^„ 

where c = This imphes that, if there exists a constant C2 > such that 

P > Pm := exp(-C2-^^), 

Ai 

then (15 ■4p is true. 

For the second term, we have 

M 

P[3/, |/| = pN, 3 J, \J\ = p'N, J2 •^2 n > 12S\J, I)] 

(1=2 

M 

I:\I\=pN J:\J\=p'N M=2 

1 72C1A1IJI 



< 



^ ^ . 2A I Me{J,J} 

I:\I\=pN J:\J\=p'N \ ^{-^'-f} 72CiAi|J| 

where = Xi{E{J, I}) is the largest eigenvalue of the graph determined by 

the undirected edges in E{J,I}. Using Lemma [4. 3 [ we get 

A{j,/} < 2[a/pp'Ai + k], 

and e{J, /} < (pp'Ai + ^/pf7hi)N, which implies 



M 



I2C1 p'N 



P[Y,S^iJ)>l2Ci\i\J\]<exp 

There are (^) (|^|) ways to choose the sets / and J and 

< exp((/i(p) + h{p')N) < exp{2h{p)n 




N\ /N 

as we assume that p' < p < 1/2. These considerations yield the condition 
72C1 p' 



> 2/i(p), 



'Ai 72C1A1 72Ci(Ai) 

which is true if 

72^1-^ > 8h{p), 72^1-^ > Ship), > Ship), 

and 

^f' r- > 16plog(-) > 8Mp). 

2 Mk p_ n 

72Ci(Ai)2^' ■ 
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From here we obtain the four conditions 

p' > c'pKp)\ p' > c^h{p), p' > c'^pKp) 

Ai Ai 



and 



32 ^^A nl - 16 



where C = -^jr and C — -, — 
For the third term, we have 



M 



P[3I, \I\ = pN,3J, \J\ = p'N,J2s^{J,J) < -13S\J,I)] 



fj,=2 

AI 



^ E ^E^3W^)<-73CiAi|J|] 

J:\J\=p'N M=2 

/ /I 73C1A1IJI , 

< exp(--- A/eUJ} )^ 

J:\J\=p'N ■^{JyJ} 73C1A1IJI 

where Ajj^j} = Xi{E{J, J}) is the largest eigenvalue of the graph determined by 
the undirected edges in E{J, J}. Using Lemma [4. 3[ we have 

A{j,j} < 2p'Xi + 2k 

and e{ J, J} < {p'Xi + k)p'N, and we obtain as for the previous terms 

1 73C1A1IJI \ ^ f -fsCi p'N 



Ox I Me{J,J} - ^^^t^ \ 2 f o I ^ 1 nM ^ I 

There are (|^|) ways to choose the set J. From this, we get the condition 

V 73C1A1/ Aip' 

which is true if 

h{p') < C and h{p') < Cp' — , where C = (5.5) 

^ ^ 73C1A1 ) 

As we assume that M < cAi, there exists a pi (73,6*1) G (0,0.1), such that the 
first inequality in (15. 5p is true if p' < pi. 

Using the bound h{p') < — 2p'log(p') again, we get that there exists c > such 
that the second condition in (15. 5p is true if 

/ / -^1 N 

p > exp(— c — ). 
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For the fourth term, we have 

M 

P[3I, \I\ = pN, 3J, I J| = p'N, S!:{J, I) > 14S\J, I)] 

M 

I:\I\=pN J:\J\=piN ii=1 

< V V exDf-- ^^^^^^I'^l \ 

- 2\ Me{JnJ,JnJ} ^' 

I:\I\=pN J:\J\=p'N '^JCM + 74C1A1IJI 

where Ajn/ = Ai(-E{Jn/, Jfl/}) is the largest eigenvalue of the graph determined 
by the undirected edges in E{J fl /, J fl /}. 

Using Lemma 14.31 and assuming that p' < p, we have Xjni < 2p'Ai + 2k, and 
e{ Jn J, Jn J} < (p'Ai + n)p'N, which are the same bounds as for the third term. 
There are (^) (^^^^ ways to choose the sets / and J and using again 

'N\ ( N\ 

< exp{{h{p) + h{p')N) < exp{2h{p)N), 



\I\J\\Jl 

we finally arrive at the same conditions as for the third term, with possibly a 
different constant C. 

Finally, the different conditions can be summarized as: 

p' > C2p po> p> p > pm, P > C^ph{pf, 
M 

P' > C^hip), p' > C'^phip) and p' > pd^,log{^-)r\ 
Ai Ai (Ai)^ p 

Finally, taking into account all the conditions, we get that (15. ip is true if we 
choose 

/(p) =max{cip {^y,ciph{p),ci^h{p),cip (-^^ log(-))^/^ p„} 
Ai Ai (Ai)^ p 

for some Ci > large enough and we see that /(p) < p if p G {pm, Po) with po 
small enough. 

□ 

In order to prove the Theorem l3.lt we will apply Lemma [5.11 repeatedly until the 
system attains an original pattern. Using 

Pm = exp(-C2 ^\,^ ), 

we get that the system can attain an original pattern, i.e. PmN < 1, only if 

Mm 

K + ^ < C2Ai/log(iV). 

Ai 

To determine the maximal number of steps the synchronous dynamics needs to 
converge, we analyze the following sequences : 
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Lemma 5.2. Let {xn)n€N, {VnJneN and {zn)n€N such that 



and 



Wo = Xo = yo = Zo = pE [exp(-;^— ), 1/e] 







Wn+l 


= cw„(— ) , 
M 






Vn+l 


M 




Mk 


Zn+1 





log(-))^/^ 



for n E N and c > 0. Let us assume that ^ > CilogN for some Ci > 1 large 
enough and that M < C2X1 for some C2 > 0. Then the sequences (wn), (xn), {Un) 
and {zn) are decreasing and there exists C3 > and 



\og{N) 

Uq > C3 max{log log A^, 



such that max{wno,Xno,yno,Zno} < ^/N. 

Proof. Let us first consider the sequence (w„). Iterating Wn+i = aWn, with a = 
c(^)^, we get trivially w„ = a"'Wo from which we deduce that w„ < as soon as 
n > Ci '"^^f^-* for some Ci > 0. 

For the sequence (x^), using h{x) < — 2xlog(x) < 2y/x for x E [0, 1/2], we have 
Xn+i < (Cxn)^^^ for some constant C > 0. Iteraring, we get x„ < (C'^xq)'-^^^^", 
from which we deduce that x^i < ;^ if n > C2loglogA^ for some C2 > 0, if xq is 
small enough. 

For the sequence using again h{x) < — 2xlog(x), we have to iterate the rela- 
tion y^+i = a log(^), with a = 2c^. If we consider |/o E [exp(— 1/a), exp(— 1)], 
the inductively defined sequence yn+i = giVn) is decreasing and converges to 
exp(— 1/a) since the function g{x) = — axlog(x) is increasing on the interval 
[exp(— 1/a), exp(— 1)], yi < yo and exp(— 1/a) is the single fixed point of g. More- 
over, we have 



yn+2 = a2y„l0g(^)(l0g(^) + log(i) + log(j^)) 

<aVlog(^)(log(^) + log(i)), 
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if l/n < By iteration, if we set b = log( 



min{p,a} ' ' 



we get similarly for all n G N, 



n-l r 



i=0 



1 1 

log(— ) + zlog(-) 



< {abr yo n\ 



T) 

< c^{ab-Y^ 

^ 1 
= C3exp(?T,(log(a) + log(6) + log(ra) - 1) + - log(?7,)) 

< C3exp(-C4 logAr(l + o(l))), 

for some C3 > 0, if n = C4 log(A^) / (— log a — log log A^). In particular, this justifies 
the hypothesis — > C5 log N for some C5 > 1 large enough. We therefore see that 



there exists some ce > such that e < Dn < for 



n = cq 



log(- 



-) 



The third sequence can be rewritten as Zn+i = a^n(log 7-)^^'^! with a = c(|^^)^^^. 
With the same technique as for we get that the sequence (zn) converges to 
exp(-l/a3/2) and Zn<l/N if 



n > C7log(A^)/log( 



MKlog(A^)'' 



This proves the lemma. 



□ 



The combination of the previous considerations and Lemma 15.21 then yields the 
Theorem 13.11 



Appendix A. On the degrees of the Erdos-Renyi graph 

To prove the corollary 13.31 , we need to estimate the minimum and the maximum 
degrees of a typical Erdos-Renyi graph G{N,p). The following result could not 
be found in the literature. We prove in this appendix : 

Lemma A.l. If G be is chosen randomly according to the model G{N,p), then, 
if p ^ ^-^j^, for a set of realizations of G the probability of which converges to 
one as N 00, we have m = (1 + o{l))Np and 6 = {1 + o{l))Np. 

Proof. Let G chosen randomly according to the model G{N,p). The law of the 
degree di of an arbitrary vertex i of G is the binomial distribution B{N,p). Hence 
using the exponential Markov inequality, we arrive at the following bound : for 
p < a < 1, and A > 1, 

P[di > a A] < exp{-NH{a,p)), 
where H is the relative entropy or Kullback-Leibler information 

H{a,p) = alog(-) + (1 - a) log(^— ^). 
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If we now set m = maxj di as above, we obtain 

TV 

P[m > aN] < ^P[rfi > aN] < N exp{-NH{a,p)). 

i=l 

If we choose a = (1 + e)p, for some e > such that a < 1, we therefore get 

P[m > {l+e)pN] < N exp (^-N{p{l + e) log(l + 5) + (1 - (1 + e)p) log( ^ ~ ^^^^ )) 

Moreover we have (1 — (1 + e)p) \og{^—j^^) > —pe. 

Indeed, if we set g = 1 — p and m = 1 — (1 + e)p = q — ep, the last inequality is 
equivalent to log(g/M) < q/u — 1 which is true since q/u > 1. Thus 

P[m > {l+e)pN] < N exp{-Np{{l+e)log{l+e)-e))) < N exp{-Np—{l+o{l))), 



/logiV 

if we suppose that e = o(l) as — ?■ 00. Choosing e = 2\ — — — , we have e = o{l] 



pN 



for p > and 



N ' 

P[m > (1 + e)pN] ^0 as iV -> 00. 

Moreover, we have m > Ai and Ai = (1 + o(l))pA^(with probability converging 
to 1 as — 00), which gives the result for m. 

Now, if we set 6 := minjdj, we want to prove that P[6 > (1 + e')pN] — 1, as 
— 00, for some e' = o(l). We consider the complementary graph, i.e. the 
random graph G, such that exactly those edges are missing in a realization of G, 
that occur in the corresponding realization of the original random graph G. Now 
the maximum degree m of G and the minimum degree 5 of G are linked via the 
relation S = N — 1 — m. 

As G is chosen randomly according to the model G{N, 1 — p), we have 
P[m > (1 + - p)N] < Nexp{-NH{{1 + e){l - p), 1 - p)), 
for all e > such that (1 + e){l - p) < 1. Now 

H{{l + e){l-p),l-p) = {l+e){l-p)\og{l+e) + {p-e+pe) log(l-^^^^). 

p 

If we suppose that e = o(l) and e -C using the inequality log(l — x) > 
—X — — x^ for X G (0, 1/2) to bound the last term, we obtain the estimate 

H{{1 + -p), 1 -p) > -p)- C{e + |) + 0{pe')), 



for some C > and 



P[m > (1 + - p)N] < exp(-A^^((l -p)- C{e + -) + 0{pe^)) + \og{N)). 

2p p 
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There exists some c > such that if we choose e = y c{i~p) n ' 

P[m >{l + s){l- p)N] < exp(-ciV— (1 - p) + log(A^)) ^ 0, 

Zp 

under the conditions p > M and 1 - p > (M)i/3_ 

Finally we get 5 > iV - 1 - (1 + £:)(! - p)iV = (1 + o{l))Np, which is the result 
under these two conditions. 

Eventually, we will extend this result for all p such that p— )■!, asA^— )-+oo. As 
previously, using the exponential Markov inequality, we get the following bound : 
for < 6 < p < 1, and > 1, 

P[di < bN] < exp{-NH{b,p)). 

We set p = 1 — a^r and b = 1 — bjy, for some strictly positive sequences {a^) and 
(^at) such that Cat + — )■ 0, as A^ — j- oo, ^ bN, and we can restrict to the 
case ttN < {c^^jf^y^^ for some c > 0. We get 

P[6 < bN] < Nexp{-N{{l - b^) log(^) + b^ log(^))) 
< exp(-A^(6^ log(^) - 2bN) + log(A^)) 



So, we need to choose b]\f such that 

6^ log(AO 

67V log — > r^. 

We have 

if we choose for instance b^ = C-^j^)'^ with 7 G (0, 1/3). 

Finally, we get for all p — )■ 1 that 6 > [1 — bN)N = (1 + o{l))Np, with probability 
converging to 1 as A^ — 00. □ 
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